Skip to content

feat: Adds a vllm backend #122

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 29 commits into from
Oct 29, 2025
Merged

feat: Adds a vllm backend #122

merged 29 commits into from
Oct 29, 2025
+2,602 −406

Conversation

@guicho271828
Copy link
Contributor

basic VLLM backend without tool and alora support.

  • Tests are copied from huggingface, failing results are optional ones as follows
=================================================== short test summary info ===================================================
FAILED test/backends/test_vllm.py::test_multiturn - assert 'β' in "\nokay, let's see. the user asked to take the result of"
FAILED test/backends/test_vllm.py::test_generate_from_raw_with_format - AssertionError: formatting directive failed for { "n...
========================================== 2 failed, 5 passed, 3 warnings in 14.99s ===========================================
  • Requires VLLM_V1=0 to be set when running.

@mergify
Copy link

mergify bot commented Sep 5, 2025 •
edited
Loading

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert|release)(?:\(.+\))?:

@guicho271828 guicho271828 force-pushed the masa/vllm-backend branch 6 times, most recently from ba3bb52 to 300c148 Compare September 5, 2025 00:57
jakelorocco
jakelorocco reviewed Sep 5, 2025
View reviewed changes
@jakelorocco jakelorocco changed the title Masa/vllm backend feat(vllm): Masa/vllm backend Sep 8, 2025
@nrfulton nrfulton changed the title feat(vllm): Masa/vllm backend feat: Adds a vllm backend Sep 10, 2025
@guicho271828 guicho271828 force-pushed the masa/vllm-backend branch 3 times, most recently from 93a191a to 69d3be8 Compare September 10, 2025 22:17
jakelorocco
jakelorocco reviewed Sep 17, 2025
View reviewed changes
jakelorocco
jakelorocco requested changes Sep 17, 2025
View reviewed changes
Copy link
Contributor

@jakelorocco jakelorocco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks mostly good to me; I left a few comments and looks like there's some pre-commit checks failing as well

jakelorocco
jakelorocco reviewed Sep 23, 2025
View reviewed changes
Copy link
Contributor

@jakelorocco jakelorocco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this looks good. There will be conflicts / changes required based on my async PR: #137. I'm happy to make those commits though once the async stuff gets merged and this branch gets rebased.

@guicho271828 guicho271828 force-pushed the masa/vllm-backend branch 5 times, most recently from e84ff46 to 048e90d Compare September 24, 2025 18:53
@guicho271828 guicho271828 force-pushed the masa/vllm-backend branch 4 times, most recently from 61d6384 to 291b459 Compare October 1, 2025 18:35
@guicho271828
Copy link
Contributor Author

async support is already implemented

guicho271828 and others added 18 commits October 21, 2025 18:59
@guicho271828
fix(vllm): added vllm optional dep in pyproject.toml
5808d95 
Signed-off-by: Masataro Asai <guicho2.71828@gmail.com>
@guicho271828
feat(vllm test): copied from huggingface
872c529 
Signed-off-by: Masataro Asai <guicho2.71828@gmail.com>
@guicho271828
fix(vllm test): implemented the test
12df402 
Signed-off-by: Masataro Asai <guicho2.71828@gmail.com>
@guicho271828
test: require V0 in vllm test
015c62b 
Signed-off-by: Masataro Asai <guicho2.71828@gmail.com>
@guicho271828
refactor: ctx to chat conversion function
003e766
@guicho271828
refactor: use_alora function
9a5f557
@guicho271828
refactor: moved _extract_model_tool_requests to mellea.backends.utils
f586128
@guicho271828
feat(vllm): added tool calls
3571098
@guicho271828
test(tools): run test with mistral
cd339c7
@guicho271828
fix(vllm): rename model_options -> engine_args
e8a69f0
@guicho271828
fix(vllm): use FancyLogger
a3ee501
@guicho271828
fix(vllm): ignore type checking for vllm and msgspec
d25579a
@guicho271828
fix(vllm): fixed the backend name in the log
fd1b3a4
@guicho271828
feat(vllm): asynchronous call support
2a12031
@guicho271828
test(vllm): asynchronous call support
356dfd5
@guicho271828
fix(vllm): avoid unnecessary incremental processing in non-streaming …
39ad431 
…mode
@guicho271828
fix(vllm): fix for the new return format
7452a1f
@guicho271828
fix(vllm): fixed vllm test for the new contexts
9daa4e2
jakelorocco
jakelorocco reviewed Oct 22, 2025
View reviewed changes
@nrfulton
Copy link
Contributor

I just merged main.

If tests do not predictably pass in the CI/CD pipeline, we should mark those as non-cicd tests (ask @avinash2692 if you need a pointer on how to do this).

Please run the full test suite locally after the main merge and ensure all tests are passing locally.

Other than that, LGTM.

@guicho271828 guicho271828 merged commit 21908e5 into generative-computing:main Oct 29, 2025
2 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jakelorocco jakelorocco jakelorocco approved these changes

@nrfulton nrfulton Awaiting requested review from nrfulton

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@guicho271828 @nrfulton @jakelorocco